Maximum Information Transfer in Feedforward Neural Networks

The principle of maximum information preservation has been successfully used to derive learning algorithms for self-organizing neural networks. In this paper, we state and apply the corresponding principle for supervised networks: the principle of minimum information loss. We do not propose a new learning algorithm, but rather a pruning algorithm which works to achieve minimum information loss in the face of noise at the input. We also explore the implications of adding noise to the problem of maximum information preservation with nonlinear neurons.